Report on the First NLG Challenge on Generating Instructions in Virtual Environments (GIVE)
نویسندگان
چکیده
We describe the first installment of the Challenge on Generating Instructions in Virtual Environments (GIVE), a new shared task for the NLG community. We motivate the design of the challenge, describe how we carried it out, and discuss the results of the system evaluation.
منابع مشابه
Report on the Second NLG Challenge on Generating Instructions in Virtual Environments (GIVE-2)
We describe the second installment of the Challenge on Generating Instructions in Virtual Environments (GIVE-2), a shared task for the NLG community which took place in 2009-10. We evaluated seven NLG systems by connecting them to 1825 users over the Internet, and report the results of this evaluation in terms of objective and subjective measures.
متن کاملReport on the Second Second Challenge on Generating Instructions in Virtual Environments (GIVE-2.5)
GIVE-2.5 evaluates eight natural language generation (NLG) systems that guide human users through solving a task in a virtual environment. The data is collected via the Internet, and to date, 536 interactions of subjects with one of the NLG systems have been recorded. The systems are compared using both task performance measures and subjective ratings by human users.
متن کاملThe Heidelberg GIVE-2 System
This paper describes a NLG system, developed by students at Heidelberg University, that generates instructions that direct users through virtual environments. The NLG architecture is implemented as a modular pipeline consisting of elementary components. The system has been developed for participation in the 2nd installment of the Generation Instructions in Virtual Environment (GIVE) challenge.
متن کاملGenerating Instructions in Virtual Environments (GIVE): A Challenge and an Evaluation Testbed for NLG
Would it be helpful or detrimental for the field of NLG to have a generally accepted competition? Competitions have definitely advanced the state of the art in some fields of NLP, but the benefits sometimes come at the price of over-competitiveness, and there is a danger of overfitting systems to the concrete evaluation metrics. Moreover, it has been argued that there are intrinsic difficulties...
متن کاملGenerating Instructions in a 3D Game Environment: Efficiency or Entertainment?
The GIVE Challenge was designed for the evaluation of natural language generation (NLG) systems. It involved the automatic generation of instructions for users in a 3D environment. In this paper we introduce two NLG systems that we developed for this challenge. One system focused on generating optimally helpful instructions while the other focused on entertainment. We used the data gathered in ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2009